Learning Visual Models for Lip Reading

This chapter describes learning techniques that are the basis of a "visual speech recognition" or "lipreading" system 1 • Model-based vision systems currently have the best performance for many visual recognition tasks. For geometrically simple domains, models can sometimes be constructed by hand using CAD-like tools. Such models are difficult and expensive to construct, however, and are inadequate for more complex domains. To do model-based lipreading, we would like a parameterized model of the com­ plex "space of lip configurations". Rather than building such a model by hand, our approach is to have the system itself build it using machine learning. The system is given a collection of training images which it uses to automatically construct the models that are later used in recognition. There are several phases of processing involved in our system. Ulti­ mately, the recognition of the time sequence of images is performed using Hidden Markov Model technology similar to that used in speech recogni­ tion. Unlike speech recognition, however, there are extra phases to find,